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Preface 


The Quality Assurance Framework was originally drafted in 1997 to describe the 
measures Statistics Canada had in place to manage the quality of the data it produced. 
The impetus for its preparation came from a then imminent audit by the Office of the 
Auditor-General (OAG) on Statistics Canada’s management of the quality of statistics. 
That audit was subsequently carried out and published by the OAG in April 1999. A 
subsequent report on Statistics Canada’s implementation of the recommendations of that 
audit was published by the OAG in December 2001. 


The present version of the Quality Assurance Framework updates the 1997 version. It 
retains the definition of data quality, and its six dimensions, as adopted in 1997. But it 
links the various measures that we use for quality management more closely to each of 
these six dimensions. It also incorporates updated references to policies, procedures and 
guidelines that have changed since 1997. 


The six dimensions of data quality as described in the Quality Assurance Framework 
have been adopted as the framework for reporting on data quality by programs in their 
Biennial and Quadrennial Program Reports, and by the Agency in its annual 
Departmental Performance Report to Parliament. 


This is a descriptive document intended for reference and training purposes. It does not 
impose any new policies or practices on programs, but situates existing policies or 
practices within a common quality management framework. The various measures 
described in the document will not necessarily apply uniformly to every program. Outside 
those requirements laid down in policies, it remains a responsibility of program managers 
to determine which measures should be applied in their programs. In addition, the Quality 
Assurance Framework will need to evolve as the Agency’s programs, methodology and 
technology evolve. 


Enquiries about the Quality Assurance Framework should be addressed to “Contact Us” 
via our web site (www.statcan.ca), Toll-free general enquiries line (1 800 263-1136), 
National TTY line - teletype machine (1 800 363-7629) or Toll-free fax number (1 877 
287-4369). 
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Statistics Canada’s Quality Assurance Framework 


1. Introduction 


Statistics Canada’s product is information. Confidence in the quality of that information 
is key to its survival. If its information becomes suspect, the credibility of the Agency is 
called into question and its reputation as an independent, objective source of trustworthy 
information is undermined. The management of quality must therefore play a central role 
within the overall management of the Agency. This Quality Assurance Framework 
describes the approaches that Statistics Canada takes to the management of quality. 


Information quality is only one of 
several critical issues that Statistics 
Canada has to manage to be 
effective. Of equal importance are 
maintaining a reputation for 
objectivity and impartiality, 
respecting privacy and 
confidentiality, sound financial 
management, and staff development 
and training. This document focuses 
on information quality and covers 
these other issues only to the extent 
that they impinge on information 
quality. 


Statistics Canada’s mandate is to serve the 
information needs of all sectors of Canadian 
society, including all levels of government. In 
addition to meeting its legislated data 
obligations, the Agency strives to develop and 
disseminate reliable and objective information 
that satisfies and anticipates critical needs - 
information that sheds light on major public 
policy issues and on the processes underlying 
social, economic and environmental phenomena 
in Canada. The Agency aims to promote the 
economic and social well-being of the country 
by providing a factual basis for better decision- 
making by governments, businesses, institutions 


Statistics Canada defines the quality 
and individuals. 


of information in terms of its fitness 
for use. This is a multidimensional 
concept embracing both the relevance of information to users’ needs, and characteristics 
of the information such as accuracy, timeliness, accessibility, interpretability and 
coherence that affect how it can be used. 


Most operations and functions of the Agency have an impact on the quality of the 
Agency’s information. The management of quality is therefore an integral part of the 
management of every program, and an important component of corporate management. It 
is not a separate management function, but an aspect of the management of the Agency 
that has to be addressed across all programs in the same way as, for example, financial 
management, human resource management, the management of respondent relations, or 
the management of data holdings. 


A significant feature of the management of quality is the balancing of quality objectives 
against the constraints of financial and human resources, the goodwill of respondents in 


providing source data, and competing demands for greater quantities of information. The 
management of quality is not the maximization of quality at all costs, but the achievement 
of an appropriate balance between the quantity and quality of information yielded by the 
Agency’s program and the resources available. Within individual programs the challenge 
is to make the appropriate trade-offs between the evolving needs of clients, costs, 
respondent burden, and the various dimensions of quality. 


Success in the management of 
quality requires a shared concern for, | Statistics Canada strives to build quality into all 


and pride in, quality among its programs and products. The quality of its 
employees and managers at all levels | official statistics is founded on the use of sound 
of Statistics Canada. Acceptable scientific methods, adapted over time to 
quality is not achieved through changing client needs, to budgetary 

managerial edicts but through circumstances, to the changing reality that the 


attention to client needs coupled Agency aims to measure, and to the capacity of 
with the sound application of respondents to supply source data. 

knowledge and expertise by 
employees at many levels. Policies 
and programs aimed at ensuring the 
availability of a motivated and competent workforce at all times are a crucial supporting 
element of the Agency’s management of quality. 


The next two sections expand on the definition of information quality and its six 
dimensions as well as the framework under which it is managed. The following six 
sections describe the processes in place for managing each of these dimensions. Section 
10 integrates some elements of the various processes described, and adds some additional 
corporate processes that complete the management of quality. 


2. Defining Information Quality 


Statistics Canada defines the information quality of its statistical outputs to reflect their 
fitness for use by clients. To operationalize this definition it has identified six dimensions 
of information quality as described in Table 1. 


Table 1 


Relevance 


Accuracy 


Timeliness 


Accessibility 


Interpretability 


Coherence 


The Six Dimensions of Information Quality 


The relevance of statistical information reflects the degree to which it meets the real needs of clients. It is 
concerned with whether the available information sheds light on the issues of most importance to users. 
Assessing relevance is a subjective matter dependent upon the varying needs of users. The Agency’s challenge 
is to weigh and balance the conflicting needs of current and potential users to produce a program that goes as 
far as possible in satisfying the most important needs within given resource constraints. 


The accuracy of statistical information is the degree to which the information correctly describes the 
phenomena it was designed to measure. It is usually characterized in terms of error in statistical estimates and is 
traditionally decomposed into bias (systematic error) and variance (random error) components. It may also be 
described in terms of the major sources of error that potentially cause inaccuracy (e.g., coverage, sampling, 
nonresponse, response). 


The timeliness of statistical information refers to the delay between the reference point (or the end of the 
reference period) to which the information pertains, and the date on which the information becomes available. It 
is typically involved in a trade-off against accuracy. The timeliness of information will influence its relevance. 


The accessibility of statistical information refers to the ease with which it can be obtained from the Agency. 
This includes the ease with which the existence of information can be ascertained, as well as the suitability of 
the form or medium through which the information can be accessed. The cost of the information may also be an 
aspect of accessibility for some users. 


The interpretability of statistical information reflects the availability of the supplementary information and 
metadata necessary to interpret and utilize it appropriately. This information normally covers the underlying 
concepts, variables and classifications used, the methodology of data collection and processing, and indications 
of the accuracy of the statistical information. 


The coherence of statistical information reflects the degree to which it can be successfully brought together 
with other statistical information within a broad analytic framework and over time. The use of standard 
concepts, classifications and target populations promotes coherence, as does the use of common methodology 
across surveys. Coherence does not necessarily imply full numerical consistency. 


These dimensions of quality are overlapping and interrelated. However, there is no 
effective model for bringing together all of these characteristics of quality into a single 
indicator. Every dimension has to be adequately managed if information is to be fit for 


use; failure in any one dimension will impair or destroy the usefulness of the information. 


Achieving an acceptable level of quality is the result of addressing, managing, and 
balancing over time, the various dimensions of quality, with due attention to program 
objectives, the major uses of the information, costs, respondent burden, and other factors 
that may affect information quality or user expectations. Actions taken to address one 
dimension of quality may affect other dimensions, often in ways that cannot be fully 
predicted. Decisions and actions aimed at achieving an appropriate balance of quality 
dimensions and other factors are based on knowledge, experience, reviews, feedback, 
consultation and, inevitably, judgement. 


Ji Management Framework 


The management of quality at Statistics Canada occurs within a matrix management 
framework — project management operating within a functional organization. The 
Agency is functionally organized into seven Fields. Four of these are primarily 
responsible for statistical programs of data production and analysis in various subject- 
matter areas. The other three Fields are primarily involved in the provision of 
infrastructure and services to be used by the statistical programs. The typical statistical 
program will be managed by one of the subject-matter divisions but will draw heavily on 
the resources of service areas for inputs to their programs, particularly for collection and 
processing operations, for informatics support, for statistical methodology, and for 
marketing and dissemination support. 


The design or redesign of a statistical program normally takes place within a project 
management structure in which the sponsoring program area and the involved service 
areas all participate. It is within such a project team that the many decisions and trade- 
offs necessary to ensure an appropriate balance between concern for quality and 
considerations of cost and response burden are made. It is the responsibility of functional 
organizations (divisions) to ensure that project teams are adequately staffed with people 
able to speak with expertise and authority for their functional area while being sensitive 
to the need to weigh competing pressures in order to reach a project team consensus. 
Projects are normally guided by a more senior Steering Committee that includes 
managers from each of the major participating areas. This Committee provides overall 
guidance, broad budgetary and design parameters, and helps to ensure that appropriate 
resources are available to the project. It also provides a forum for resolving any issues 
that cannot be satisfactorily resolved at the project team level. 


The use of an interdisciplinary project team approach, supported by the functional 
organization, is important in ensuring that quality considerations receive appropriate 
attention during design, implementation and assessment. Subject-matter staff bring 
knowledge of content, client needs, and relevance. Methodologists bring their expertise 
on statistical methods and data quality trade-offs, especially with respect to accuracy, 
timeliness and cost. Operations experts bring experience in operational methods, and 
concerns for practicality, efficiency, field staff and respondents. The systems expert 
brings a systems view, and knowledge of technology standards and tools to the design. 
Together they have to balance the conflicting pressures to develop an optimal design. The 
fact that each is a part of a specialized functional organization, from which they can call 
on a variety of more specialized and management resources when warranted, helps in 
resolving both technical challenges and conflicts arising in a project. 


4. Managing Relevance 


The management of relevance embraces those processes that lead to the determination of 
what information the Agency produces and the level of resources to be devoted to each 
program. It deals essentially with the translation of user needs into program approval and 
budgetary decisions within the Agency. The processes that are used to assure relevance 
also permit basic monitoring of other elements of quality and correspondingly to assess 
user requirements in these other dimensions. 


To fulfill its mandate it is paramount that the Agency’s programs and outputs properly 
and continuously reflect the country’s most important information needs. Since these 
needs evolve over time, a process for continuously reviewing programs in the light of 
client needs and making necessary adjustments is essential. 


There are constraints on adjustment. It has been estimated that more than 90% of the 
Agency’s budgetary resources are devoted to ongoing programs that are non-discretionary 
at a given point in time. These programs serve the information needs of a broad clientele 
through provision of basic information on Canadian society and the Canadian economy, 
and they meet the legislative and regulatory needs specified in approximately two dozen 
Acts of Parliament. The Agency’s response to newly emerging information needs must 
therefore be found through savings within these non-discretionary programs that do not 
imperil their outputs, through redirection of resources within the discretionary 
component, or through persuading clients (particularly federal government clients) to 
finance worthy additions to the national database. 


A second constraint on adjustment is the interdependency between different programs. In 
many cases information from one program feeds another (e.g. retail sales information 
feeds into GDP calculations, vital statistics are used in population estimates) so that the 
impact of adjustments in one program on others has to be considered. 


While taking into account these constraints, the Agency has put in place processes that 
monitor the relevance of its existing programs, that identify new or emerging information 
gaps that the current program is not filling, and that lead to decisions on program change 
or new funding initiatives aimed at strengthening the relevance of the Agency’s overall 
program. These processes can be described under three broad headings: client and 
stakeholder feedback mechanisms; program review; and data analysis. A fourth 
process, priority-setting and planning translates the information from these three 
processes into program and budget decisions. 


4.1 Client and Stakeholder Feedback Mechanisms 


Feedback mechanisms serve to maintain awareness of the issues of interest to each major 
client and stakeholder group, and the information needs likely to flow from these issues. 
They also obtain feedback from current users of our products on their level of 


satisfaction, and identify potential new markets for information. The principal 
mechanisms include: 


the National Statistics Council - provides overall advice on policies and 
priorities for statistical programs; 

fifteen Professional Advisory Committees in major subject areas - regularly 
review Statistical programs and plans to amend them; 

senior bilateral arrangements with key federal departments and agencies (e.g., 
Human Resources Development, Bank of Canada, Health Canada, Canadian 
Customs and Revenue Agency, Agriculture Canada, Transport Canada) - serve 
to keep abreast of emerging issues among our principal Federal government 
users and suppliers; 

participation of the Chief Statistician in policy and program discussions with 
other Deputy Ministers - keeps Agency management aware of current and 
emerging issues within the Federal government; 

involvement of senior Agency staff in the Policy Research initiative of the 
Federal government; 

the Federal-Provincial Consultative Council on Statistical Policy and its 
subsidiary committees - provide a means of liaising with provincial 
governments on their statistical requirements; 

special liaison and consultation arrangements with Federal and Provincial 
officials in areas of primary provincial jurisdiction (health, justice, education); 
periodic liaison with business associations and labour unions - help to 
understand information needs and reporting preferences in the business sector; 
ad hoc consultations with interested groups on particular programs (e.g. on 
Census content) - provide input to the design of these programs; 

bilateral liaison with foreign statistical agencies, and multilateral liaison 
through international organizations (e.g., OECD, Conference of European 
Statisticians, International Statistical Institute) - identify information needs 
emerging in other jurisdictions; 

user feedback through Advisory Services - provides information on the 
strengths and weaknesses of existing products and gaps in our product line; 
market research, and monitoring of product sales and requests - identify 
current product use and demand for new products; 

bilateral and multilateral discussions with potential clients - to identify cost- 
recovery projects. 


Together these mechanisms provide a flow of information on satisfaction with current 
products, gaps in our current set of products, and information needs likely to emerge in 


the future. 


4.2. Program Review 


In addition to the regular flow of information coming from liaison mechanisms, 
programs are periodically reviewed to assess whether they are meeting user needs. Every 


two years each program is required to produce a program report documenting its 
performance and laying out its future direction and proposals for change. These reports 
follow the structure of this Framework in reporting on the management of quality. 


Every fourth year, each program must supplement its biennial report with a more strategic 
review of its relevance and direction, including the results of consultation with its clients 
and stakeholders. These quadrennial reports consolidate and analyse the feedback 
obtained from clients and may present proposals for addressing any identified program 
weaknesses. 


Both the biennial and quadrennial reports are reviewed by senior management, with 
written feedback being provided to the programs. The quadrennial reports are presented 
and discussed at the Agency’s Corporate Planning Committee, and any proposals put 
forward are considered in the context of the Agency’s planning process (see 4.4 below). 


In addition to these regular reviews, ad hoc reviews or audits may be undertaken on 
particular programs or issues in response to feedback arising from key users, problems 
identified within programs, proposals for program change, or sudden changes in the 
external environment. 


4.3 Data Analysis 


Data analysis serves several purposes in the management of quality, including an 
important role in maintaining relevance. While its primary purposes may be to advance 
understanding and to discover further insights from existing outputs, it also provides a 
valuable source of feedback on the adequacy and completeness of our own data. By 
identifying questions the data cannot answer it pinpoints gaps and weaknesses in our data 
holdings. The development of certain longitudinal surveys, several record linkage 
activities, the creation of a metadata base, harmonized calibration, and attention to data 
integration and standardization of concepts are among the initiatives that can be 
attributed, at least in part, in response to obstacles faced in undertaking analysis. 


The peer review mechanisms for analytical work, including articles published in the 
Agency’s flagship publications - Perspectives on Labour and Income, Canadian Social 
Trends and Canadian Economic Observer, provide critical and expert review of analytical 
work and another source of feedback on our programs. 


The use of analytic frameworks such as the System of National Accounts to integrate and 
reconcile data coming from different sources is an important element in identifying gaps 
and weaknesses in our data. The development of other analytic frameworks, that would 
allow similar integration and reconciliation for other domains of subject-matter, 
continues. 


Specific issues of relevance identified through data analysis may lead to content changes 
in existing programs. Findings related to other dimensions of quality (especially accuracy 
and coherence) may also emerge from analysis (see section 10.3) 


An active program of analysis is encouraged and nurtured through several mechanisms: 


= the centralized analytical programs in the Analytical Studies Branch; 

= decentralized analysis in subject-matter divisions; 

* the Doctoral and Post-doctoral Fellowship programs for external analysts; 

= joint analytic work with external authors, often academics; 

= subject-matter data review and reconciliation committees (e.g., the Economic 
Forum for the System of National Accounts and its “feeder” surveys); 

= the Research Data Centres, where academic researchers produce reports for 
Statistics Canada; 

* contracts with external analysts to produce analytical reports for Statistics 
Canada. 


4.4 The Planning Process 


Statistics Canada’s long-term planning process (LTP) provides the framework for 
deciding what changes will be made to the Agency’s program from one year to the next. 
The process involves all managers in the Agency and takes as one of its major inputs the 
information and intelligence on user needs, program weaknesses, and information gaps 
that managers at various levels have gleaned through the mechanisms described above, or 
by other means. The process is described in detail in other documents (Statistics Canada, 
1998b). The features of the process that are particularly important to managing relevance 
are the following: 


= an annual strategic planning conference to identify major corporate priorities 
to be addressed in the coming LTP round; 

= arequirement for program managers to identify their lowest priority program 
activities for possible reduction or cancellation and to propose efficiencies; 

= an invitation to program managers to submit new initiatives that would 
respond to user needs, especially in areas identified as corporate priorities; 

= areview and screening of proposals by program Syndicates headed by 
Assistant Chief Statisticians; 

= an independent review of major systems proposals to ensure that they follow 
appropriate project management and design principles; 

= a senior management meeting of Directors General and above to review and 
critique all proposals presented by Syndicates; 

= Corporate Planning Committee decisions on all proposals based on their 
relevance to corporate priorities, the results of previous Program reviews, and 
the funds available. 


Requests to fund improvements to other aspects of quality are also submitted by program 
managers. Such opportunities can usually be undertaken with parallel gains in efficiency. 
Indeed there is a clear link between efficiency and relevance of the Agency’s overall 
program, as improvements in efficiency can be redirected to broadening the scope and of 
this program. Significant improvement in efficiency, of course, may involve program 
redesign initiated through the planning process. 


Given the limited room to maneuver within the Agency’s budget, the annual LTP process 
cannot cover full funding of all major new data collection initiatives and program 
proposals. Its funding tends to go towards non-discretionary redesigns, feasibility studies, 
seed money to evaluate new approaches to data development, and investment in 
efficiency. In some cases partial funding may be granted on the understanding that a key 
client (usually a federal department) will also contribute. For major new program 
initiatives that are clearly beyond the financial means of the Agency to fund from its 
existing budget, external avenues of funding are pursued. Successful ventures of this kind 
require strong expressions of support, if not funding itself, from influential users - so 
success implies relevance in this case. 


In summary, the Agency uses a variety of mechanisms to keep abreast of clients’ 
information requirements, to obtain users’ views on its existing products and services, to 
seek professional advice on issues and priorities, and to identify weaknesses and gaps in 
its own data. This information provides a basis for the management judgements that have 
to be made on revising the Agency’s program. In addition to user needs and costs, 
respondent burden, public sensitivities, and the Agency’s capacity and expertise have to 
be taken into account. Judgements have to be made in the light of current public policy 
priorities as to which statistical programs are in most need of redevelopment or further 
investment. 


While decisions on program funding are the major determinant of relevance, many 
subsequent decisions during the design of programs also have an impact on the eventual 
relevance of information products. For example, questionnaire design, accuracy- 
timeliness trade-offs, and the design of statistical outputs all affect relevance. These 
considerations are dealt with in subsequent sections. 


5. Managing Accuracy 


Processes described under relevance determine which programs are going to be carried 
out, their broad objectives, and the resource parameters within which they must operate. 
Within those “program parameters” the management of accuracy requires particular 
attention during three key stages of a survey process: design, implementation, and 
assessment. These stages typically take place in a project management environment, 
outlined in Section 3, which is crucial to a proper balancing of conflicting considerations 
in the development of a statistical program. 


S.1 


Program Design 


The collection, processing and compilation of data require the use of sound statistical, 
and analytical methods and models, effective designs, instruments, operational methods 
and procedures, and efficient systems and algorithms. The accuracy achieved - as well as 
the degree of timeliness and coherence - will depend on the explicit methods put in place 
and the quality assurance processes built in to identify and control potential errors at the 
various stages of the program. While individual program managers have considerable 
flexibility in implementing specific practices and methods, standards and guidelines exist 
for some functions. Eight primary aspects of design that need to be addressed in every 
program to ensure that accuracy considerations are given due attention during design have 
been identified. 


Explicit consideration of overall trade-offs between accuracy, cost, 
timeliness and respondent burden during the design stage. The extent and 
sophistication of these considerations will depend on the size of the 
program, and the scope for different options in light of the program 
parameters. But evidence that proper consideration was given to these 
trade-offs should be visible. 


Explicit consideration of alternative sources of data, including the 
availability of existing survey data or administrative records, to minimize 
new data collection. This issue focuses on the minimization of respondent 
burden and the avoidance of unnecessary collection. 


Adequate justification for each question asked, and appropriate pre-testing 
of questions and questionnaires in each mode of collection, while also 
assuring that the set of questions asked is sufficient to achieve the 
descriptive and analytical aims of the survey. 


Assessment of the coverage of the target population by the proposed 
survey frames. 


Within overall trade-offs, proper consideration of sampling and estimation 
options and their impact on accuracy, timeliness, cost, response burden, 


and comparisons of data over time and across programs. 


Adequate measures in place for encouraging accurate response, following 
up nonresponse, and dealing with missing data. 


Proper consideration of the need for quality control and other quality 
assurance processes for all stages of collection and processing. 


Appropriate internal and external consistency checking of data with 
corresponding correction or adjustment strategies. 


10 


While these eight areas do not cover all aspects of survey design, and consideration of 
issues does not necessarily result in the “optimum” decision, evidence that these aspects 
have been seriously considered will be strongly suggestive of sound survey design. In the 
end, the strength of the survey methodology will depend on the judgements of survey 
design teams. 


Decisions on what constitutes acceptable accuracy are left to the individual program to 
determine and justify in light of its knowledge of user requirements and the 
circumstances, constraints, opportunities and objectives within which it has to work. The 
Agency’s Draft Policy on Data Quality Criteria (Statistics Canada, 1994) provides some 
guidance on the definition of acceptable accuracy levels and their implications for the 
dissemination of data. 


Whatever specific methods are applied must be within the realm of commonly accepted 
and defendable statistical practices under the given circumstances. The use of new 
technologies and innovations to improve quality and efficiency is encouraged, but must 
be adequately tested to minimize risk. Questionnaires, in particular, must be tested to 
ensure that respondents can and will be willing to provide input data of acceptable quality 
(Policy 2.8). It must be possible to monitor quality, to react effectively to unanticipated 
problems and to be able to verify or support the credibility of the results, as well as to 
understand their limitations. 


The Agency’s Quality Guidelines (Statistics Canada, 1998a) describe specific practices, 
methods and considerations that should be taken into account in designing programs, and 
indicate where formal standards or guidelines exist. 


In order to ensure that the design of statistical programs efficiently meet the objectives set 
for them a variety of means have been put in place to provide guidance and information. 
The effectiveness of these is largely a matter of their overall impact rather than of any one 
item individually: 


° the use of specialized staff for subject-matter, methodology, operations 
and systems to participate in program design; 

° Quality Guidelines to list considerations and provide guidance in design 
decisions; 

° centralized headquarters operations staff, and a regional network of field 
staff for conducting collection and processing; 

° a software use policy that identifies recommended and supported software 
for key functions; 

° specialized resource and support centres for certain functions (e.g., 
questionnaire design and testing, seasonal adjustment, data analysis); 

° standard survey frames for major populations (e.g., the Business Register, 


the LFS area sample, the Address Register); 
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° specific Agency policies and guidelines, e.g., Policy on the Development, 
Testing and Evaluation of Questionnaires (Policy 2.8), Guidelines for 
Seasonal Adjustment and Trend-Cycle Estimation (Statistics Canada, 
2000), Policy on Estimates with Future Reference Dates (Policy 2.2); 


° standard definitions of concepts, variables and classifications for common 
subject-matter areas (Policy 2.10); 

° Peer and Institutional Reviews (Policy 2.5); 

° internal cost recovery in order to reflect real costs of alternative 


approaches in making design decisions. 


While each of these mechanisms influence accuracy, they will also have an impact on 
other dimensions of quality, especially timeliness and coherence. 


5.2 Implementation 


A good design can be negated in implementation. Though a good design will contain 
built-in protection against implementation errors (through quality assurance processes, for 
example), still things can go wrong. The results of implementation depend not only on 
the specific design, but also on the instruments of implementation. These instruments 
will include the resource and material plans, the supervisory structure, the schedules, the 
operations, procedures and checks, the training, the publicity, etc., developed and 
specified during the design phase. Mechanisms for monitoring implementation should be 
built into survey processes as part of design. Two types of information are required. The 
first is information to monitor and correct, in real time, any problems arising during 
implementation. This requires a timely information system that provides managers with 
the information they need to adjust or correct problems while the survey is in progress. 
The second need is for information to assess, after the event, whether the design was 
carried out as planned, whether some aspects of the design were problematic in operation, 
and what lessons were learned from the operational standpoint to aid design in the future. 
This too requires information to be recorded during implementation (though not 
necessarily with the same fast feedback as for the first need), but it can also include 
information gleaned from post-implementation studies and debriefings of staff involved 
in implementation. 


Information pertaining directly to accuracy itself may only be a small subset of the 
information required by operational managers. But information related to costs and 
efficiency of operations is equally important to the consideration of accuracy for future 
designs. 


Statistics Canada’s management information systems form the framework that provides 
these kinds of information. In addition, the following are examples of activities that could 


be undertaken to manage and monitor implementation: 


° regular reporting and analysis of response rates and completion rates 
during the collection phase; 
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° monitoring refusal and conversion rates; 


° monitoring interviewer and respondent feedback; 

° monitoring of edit failure rates and the progress of corrective actions; 

° monitoring the results of quality control procedures during collection and 
processing; 

° monitoring of expenditures against progress; 

° development, implementation and monitoring of contingency plans. 


Where applicable, monitoring should be at various geographic levels or aggregations 
useful for each level of management, including levels suitable for supervising and 
correcting the actions of groups or individuals involved. 


5.3 Accuracy Assessment 


The third key stage of the survey process is the assessment of accuracy — what level of 
accuracy has actually been achieved? Though described last, it needs to be a 
consideration at the design stage since the measurement of accuracy often requires 
information to be recorded as the survey is taking place. 


As indicated earlier, accuracy is multidimensional and choices have to be made as to what 
are the most important indicators for each individual survey. Also each survey produces 
thousands of different estimates, so either generic methods of indicating the accuracy of 
large numbers of estimates have to be developed, or the indicators are restricted to certain 
key estimates. 


As with design, the extent and sophistication of accuracy assessment measures will 
depend on the size of the program, and on the significance of the uses of the estimates. 
Statistics Canada’s Policy on Informing Users of Data Quality and Methodology (Policy 
2.3) requires at least the following four primary areas of accuracy assessment to be 
considered in all surveys. 


1. Assessment of the coverage of the survey in comparison to a target 
population, for the population as a whole and for significant sub- 
populations. This may mean assessing the coverage of a list frame (e.g., a 
business register by industry), the coverage of a census that seeks to create 
a list of a population (e.g., the coverage of a census of population by 
province or by age and sex), or the coverage of an area sample survey in 
comparison to independent estimates of the target population (e.g., the 
difference between sample based population estimates from a household 
survey and official population estimates). 


2. Assessment of sampling error where sampling was used. Standard errors, 
or coefficients of variation, should be provided for key estimates. Methods 
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of deriving or approximating standard errors should be indicated for 
estimates not provided with explicit standard errors. 


a Nonresponse rates, or percentages of estimates imputed. The objective is 
to indicate the extent to which estimates are composed of “reported” data. 
For skewed populations (such as most business populations), nonresponse 
or imputation rates weighted by a measure of size are usually more 
informative than unweighted ones. 


4, Any other serious accuracy or consistency problems with the survey 
results. This heading allows for the possibility that problems were 
experienced with a particular aspect of a survey causing a need for caution 
in using results. For example, a widely misunderstood question might lead 
to misleading estimates for a particular variable. It also allows any serious 
inconsistencies between the results and other comparable series to be 
flagged. 


In addition, depending on the nature of the survey, attention may be needed on particular 
sources of measurement or processing error. 


The choice of how much effort to invest in measuring accuracy is a management decision 
that has to be made in the context of the usual trade-offs in survey design. But requiring 
that, at a minimum, information on the four primary aspects of accuracy be available for 
all programs ensures that attention is paid to accuracy assessment, and also provides a 
basis for monitoring some key accuracy indicators corporately. Regular measures of the 
coverage of major survey frames such as a business register or an address register also 
provide information that is important both to individual programs using these frames, and 
to Agency management. 


Measures of accuracy are an important input for Program Review (section 4.2) for 
assessing whether user requirements are being met, and for allowing appropriate analytic 
use of the data. They are also a crucial input to the management of interpretability as 
elaborated in section 8 below. 


5.4 Independent Review 


In light of the high technical content of many design issues, programs are encouraged to 
incorporate independent technical review into their design, implementation and accuracy 
assessment plans. Among options available are: 


° internal technical review committees for major programs; 


° referral of issues of technical standards, or general methods or approaches 
to the Methods and Standards Committee; 
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° referral of technical issues to the Advisory Committee on Statistical 
Methods (or to other advisory committees on specific programs); 


° review of the practices of other national statistical agencies and the 
exchange of experiences with them; 

° participation in working groups of multilateral international organizations 
addressing particular technical problems; 

° presentation of technical issues and proposed solutions for review at 
Symposia and other professional meetings; 

° use of Work-in-Progress reviews subject to the procedures laid out in the 


Policy on Statistics Canada’s Daily (Policy 3.3). 


Reviews of practices, within or across programs, may be imposed from time to time as a 
result of external or internal audits, Methods and Standards Committee initiatives, or 
other managerial decisions. 


6. Managing Timeliness 


Timeliness of information refers to the length of time between the reference point, or the 
end of the reference period, to which the information relates, and its availability to users. 
The desired timeliness of information derives from considerations of relevance — for what 
period does the information remain useful for its main purposes? The answer to this 
question varies with the rate of change of the phenomena being measured, with the 
frequency of measurement, and with the immediacy of response that users might make to 
the latest data. 


Planned timeliness is a design decision, often based on trade-offs with accuracy - are later 
but more accurate data preferable to earlier less accurate data? — and cost. Improved 
timeliness is not, therefore, an unconditional objective. However, timeliness is an 
important characteristic that should be monitored over time to warn of deterioration. It 
should be monitored across programs, to recognize extremes of tardiness, and to identify 
good practices. User expectations of timeliness are likely to heighten as they become 
accustomed to immediacy in all forms of service delivery thanks to the pervasive impact 
of technology. Unlike accuracy, timeliness can be directly observed by users who, one 
can be sure, will be monitoring it. 


To the extent that timeliness is a factor in the design and implementation of programs, the 
factors and considerations described in Section 5 apply equally here. But there are further 
measures that can be pursued for managing timeliness. 


Major information releases should have release dates announced well in advance. This 
not only helps users plan, but it also provides internal discipline and, importantly, 
undermines any potential effort by interested parties to influence or delay any particular 
release for their benefit. The achievement of planned release dates should be monitored 
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as a timeliness performance measure. Changes in planned release dates should also be 
monitored over longer periods. 


For some programs, the release of preliminary data followed by revised and final figures 
is used as a strategy for making data more timely. In such cases, the tracking of the size 
and direction of revisions can serve to assess the appropriateness of the chosen 
timeliness-accuracy trade-off. It also provides a basis for recognizing any persistent or 
predictable biases in preliminary data that could be removed through estimation. 


For ad hoc surveys and new surveys, another possible indicator of timeliness is the 
elapsed time between the commitment to undertake the survey and the release date. This 
measure reflects the responsiveness of the Agency in planning and setting up a survey as 
well as its execution after the reference date. But its interpretation must take account of 
other factors that help to determine how quickly a new survey should be in place — faster 
is not necessarily better. 


For programs that offer customized data retrieval services, the appropriate timeliness 
measure is the elapsed time between the receipt of a clear request and the delivery of the 
information to the client. Service standards should be in place for such services, and 
achievement of them monitored. 


Improvements in timeliness might be expected as new technologies are developed and as 
uses of data change. There may be an ongoing need to assess current practices through 
operational evaluations, experimentation, testing and process measurement. The nature 
of operations and how operations are organized, the quantity and mix of resources, and 
the role of manual versus automated operations require periodic review and assessment. 
Improvements in timeliness and the ability to inform users on timeliness constraints 
should lead to better management of timeliness as a key quality factor. 


ye Managing Accessibility 


Statistical information that users don’t know about, can’t locate, can’t access or can’t 
afford, is of no value to them. Accessibility of information refers to the ease with which 
users can learn of its existence, locate it, and import it into their own working 
environment. Corporate-wide dissemination policies and delivery systems determine 
most aspects of accessibility. Program managers are responsible for designing statistical 
products, choosing appropriate delivery systems and ensuring that statistical products are 
properly included within corporate catalogue systems. 


Statistics Canada’s dissemination objective is to maximize the use of the information it 
produces while ensuring that dissemination costs do not reduce the Agency’s ability to 
collect and process data in the first place. Its strategy is to make “public good’ 

information of broad interest available free of charge through several media (including 
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the press, the internet, research data centres and libraries) while charging for products and 
services that go beyond satisfying a broad public demand for basic information (Policy 
3.6). 


7.1 Product Definition and Design 


In determining what information products to offer, program managers must take careful 
account of client demands and decide whether pre-specified catalogued products or the 
provision of information on demand is justified from a cost-revenue perspective. 


Program managers are responsible for conducting market research and liaison with their 
clients to establish what products will best meet their client needs. They are also 
responsible for collecting and monitoring client feedback on the content of their products 
and on the mode of dissemination with a view to future improvement. 


Marketing Division assists these processes by: 


° making client and user profiles, as well as reports of product sales and distribution 
from the Business Information Reporting System (BIRS), available to program 
managers; 

° conducting market research studies for Statistics Canada products and services, 
and client satisfaction measurements required in divisional program reviews; and 

° providing assessments of electronic products, and support to end-users. The 


Electronic Products Help Line also captures and monitors client satisfaction, and 
provides feedback to product managers. Informing managers about the strengths 
and weaknesses of their products ensures that electronic products meet the needs 
of clients and are constantly improving. 


In the internal design of products, program managers must also ensure that the following 
policies and standards are addressed: 


° Policy 2.1 - Highlights in Publications; 

° Policy 2.3 - Informing Users of Data Quality and Methodology; 

° Standards and Guidelines on the Presentation of Data in Statistical Tables 
(Statistics Canada, 2002); and 

° Policy 2.5 - Review of Information Products. 


7.2 Dissemination 


At the corporate level, the primary dissemination vehicles include: 


° The Daily for the initial release of all data; 
° CANSIM as the repository of all publicly available data; 
° the Statistics Canada website as a primary entry point for those seeking data; and 
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° an extensive program of publications and analytical reports for specific client 
groups. 


The Internet site is to be a virtual library of all information that Statistics Canada can 
provide to the public. It should include the data released as well as information about the 
data (metadata) such as data quality statements and descriptions of the surveys underlying 
the data. The site offers extensive hyperlinks between these complementary data and 
information, which assist clients in navigating the site. 


The Government’s depository libraries program ensures that all our products are available 
to libraries across the country. The Data Liberation Initiative makes sure that universities 
have access at reasonable cost to an array of Agency products for educational and 
research purposes. A growing number of public and private-sector organizations are 
licensed to use our data to provide businesses and specialized communities with value 
added services which Statistics Canada cannot respond to directly. 


Program managers must ensure that their products are included in corporate 

dissemination vehicles as appropriate: 

° The availability of all new data sets and publications are to be announced in The 
Daily (Policy 3.3); 

° All published or publishable data should be made available through CANSIM; 


° All products and services must be registered in the Corporate Database of 
Products and Services before release (Policy 3.10); and 
° Information about data released are to be stored in the Integrated Metadata Base 


(IMDB) - see Section 8 for more on metadata. 


In addition, programs must supply data as required for corporate information products 
and services including the Statistics Canada website, the Canada Year Book, and other 
compendia. 


7.3 Needs of Analysts 


The information needs of the analytical community present some particular needs that 
have generated additional approaches to making data accessible. Analysts often require 
access to individual data records so that they can apply analytical techniques that cannot 
be performed when only pre-aggregated summaries of information are available. But the 
Statistics Act prevents the Agency from making individual records available in any form 
that might identify individuals. The following options are open to program managers to 
make their data files more accessible for analytical purposes: 


° The production of public-use microdata files that have been screened to protect 


confidentiality. Such products must be approved by the Microdata Release 
Committee; 
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° The provision of a custom retrieval service through which external analysts can 
submit special requests for retrievals from the confidential microdata base. Such 
retrievals must be screened for confidentiality before release; 


° Contracting with an external analyst to perform statistical analysis for the Agency 
under the Statistics Act. Such analysts must be sworn in under the Act; and 
° Referral of an external analyst to the Research Data Centres program (or 


equivalent) administered by the Social Sciences and Humanities Research Council 
of Canada (SSHRC). Under this program, analysts with approved projects may be 
sworn in under the Statistics Act and have access to certain files in a Research 
Data Centre (see the Guide for Researchers under Agreement with Statistics 
Canada for more information about this option); 


° Use of Sections 11 or 12 of the Statistics Act for sharing of micro-data with other 
organizations under prescribed conditions; 
° Use of Section 17 of the Statistics Act for discretionary release of certain types of 


data under prescribed conditions. 


7.4 Finding What You Want 


A major component of ensuring accessibility is providing efficient search mechanisms to 
help users find what they need. Statistics Canada provides services and tools to help its 
users in this respect. Regional Reference Centres answer client inquiries by phone, letter 
and e-mail. 


The Internet site offers an array of search and navigation tools and features that permit 
users to discover our information: 


° data and product browsers and search, by theme and subject; 
: catalogue search; 

° key word search (supported by a terminology Thesaurus); 

° The Daily; 

° CANSIM search; 

° search of the Statistics Canada’s library (BiblioCat); 

° guides to data, to search tools and to methods; and 

° index of reference documents. 


Advisory Services provides a single point of access to Statistics Canada information and 
services through a network of offices across the country. This service promotes the 
increased effective use of Statistics Canada’s data products and services. In response to 
contacts, staff: 


° help define the information requirements of the client; 
° provide data and information on the Agency’s products and services; 
° develop customized, cost-efficient data solutions; and 
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° facilitate and serve as the direct link to the rest of Statistics Canada researchers, 
analysts, consultants and other technical experts. 


The quality of this service to the public is measured on a regular service by Advisory 
Services Division. 


8. Managing Interpretability 


Statistical information that users cannot understand - or can easily misunderstand - has no 
value and may be a liability . Providing sufficient information to allow users to properly 
interpret statistical information is therefore a responsibility of the Agency. ‘Information 
about information’ has come to be known as meta-information or metadata. Managing 
interpretability is primarily concerned with the provision of metadata. 


The information needed to understand statistical data falls under three broad headings: 
(a) the concepts and classifications that underlie the data; 

(b) the methodology used to collect and compile the data; and 

(c) measures of accuracy of the data. 


These three headings cover what has been measured, how it was measured, and how well 
it was measured. Users clearly need to know what has been measured (to assess its 
relevance to their needs), how it was measured (to allow appropriate analytic methods to 
be used), and how well it was measured (to have confidence in the results). Since we can 
rarely provide a profile of all dimensions of accuracy, the description of methodology also 
serves as a surrogate indicator of accuracy: it allows the user the option of assessing 
whether the methods used were scientific, objective and carefully implemented. 


In the case of public-use micro-data files, information regarding the record layout and the 
coding/classification system to code the data on the file is an essential tool to allow users 
to understand and use the data files. Wherever possible, these files should be provided in 
a generic format, which can be read by the most popular statistical packages (SAS and 
SPSS). 


Statistics Canada’s standards and guidelines for the provision of metadata derive from the 
Policy on Informing Users of Data Quality and Methodology (Policy 2.3). It lays out 
requirements and guidelines for the provision of information on data quality and 
methodology with every statistical product. The Integrated Metadata Base (IMDB) is the 
repository used to store this information for each survey, in addition to other related 
metadata. 


Program managers are responsible for ensuring that their products meet the requirements 


of the Policy on Informing Users of Data Quality and Methodology, and for documenting 
their programs within IMDB. Particular effort is required to ensure that the information 
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provided to users is comprehensible and not obscured by our internal jargon, and to 
ensure that the information is always up-to-date. Periodic user liaison to assess the 
usefulness and adequacy of metadata is recommended. 


A further aid to Statistics Canada’s clients is interpretation of data as they are released. 
Commentary in The Daily and in associated materials focuses on the primary messages 
that the new information contains. Directed particularly at the media, such commentary 
increases the chance that at least the first level of interpretation to the public will be clear 
and correct. Conversely, Statistics Canada publicly answers or refutes serious 
misinterpretation of its data (Policy 1.2). Highlights of Publications (Policy 2.1) requires 
that all statistical publications contain a section that highlights the principal findings in 
the publication. 


9. Managing Coherence 


Coherence of statistical data includes coherence between different data items pertaining 
to the same point in time, coherence between the same data items for different points in 
time, and international coherence. Three complementary approaches are used for 
managing coherence in Statistics Canada. 


The first approach element is the development and use of standard frameworks, concepts, 
variables and classifications for all the subject-matter topics that are measured. This aims 
to ensure that the target of measurement is consistent across programs, that consistent 
terminology is used across programs (so that, for example, “educational level” means the 
same thing whether measured in a Census of population or from school records), and that 
the quantities being estimated bear known relationships to each other. The realization of 
this element is normally through the adoption and use of frameworks such as the System 
of National Accounts and standard classification systems for all major variables. The 
issue of international comparability is addressed by considering the adherence of the 
standards adopted to international standards where these exist. 


The Policy on Standards (Policy 2.10) provides the umbrella under which standards are 
approved and disseminated. At present this policy covers only a subset of the subject- 
matter areas and variables which the Agency measures. It is an ongoing objective to 
extend the coverage of this policy to other subject-matter areas as the need for common 
standards becomes apparent in each area. 


Where standards exist, programs are responsible for adhering to them or, in exceptional 
cases, seeking an exemption. Where they do not exist, programs are encouraged to 


propose the appropriate concepts and definitions that might be adopted as standards. 


The second approach aims to ensure that the process of measurement does not introduce 
inconsistency between data sources even when the quantities being measured are defined 
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in aconsistent way. The development and use of common frames, methodologies and 
systems for data collection and processing contribute to this aim. Examples of ways in 
which this approach is implemented in Statistics Canada include the following: 


° the use of a common business register as the frame for all business surveys; 

° the use of commonly formulated questions when the same variables are being 
collected in different surveys; 

° the application of “harmonized” methodologies and the embodiment of common 


methodology for survey functions (e.g. sampling, editing, seasonal adjustment) 
into software included in the Agency’s software toolbox (Policy 5.12); 


° the Quality Guidelines document to encourage consistent consideration of design 
issues across surveys; 
° the establishment of centres of expertise in certain methodologies and 


technologies to exchange experience, identify best practice, develop standards, 
and provide training; 

° reference to international codes of best practice such as the European Current Best 
Methods, or the U.N.’s Good Practices in Official Statistics. 


These first two approaches attempt to ensure that we do not build into the design or 
implementation of statistical programs any unjustified inconsistency. They do not 
guarantee perfect agreement between data coming from different sources, or eliminate 
random variations due to sampling or measurement error. 


The third approach analyses the data themselves and focuses on the comparison and 
integration of data from different sources or over time. This kind of analysis attempts to 
recognize situations where variation or inconsistency exceeds levels implied by the 
expected accuracy of the data. Such situations need to be explained by other factors and, 
where possible, corrected. Conceptual frameworks covering particular subject-matter 
areas play an important role by providing a basis for establishing coherence or 
recognizing incoherence. Some integration activities are regular and routine, e.g., the 
integration of data in the national accounts, benchmarking or calibration of sub-annual 
and annual estimates, reconciliation of survey data with administrative sources. Other 
activities are more exploratory or ad hoc. The confrontation of data from different 
sources, and their subsequent reconciliation or explanation of differences, is an activity 
that is often needed as part of pre-release review or certification of data to be published. 
Feedback from external users and analysts of data that point out coherence problems with 
current data is also an important component of coherence analysis. Some incoherence 
issues only become apparent with the passage of time and may lead to historical revisions 
of data. 
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10. Environmental Factors 


The management of the six dimensions of quality takes place in an organizational 
environment. While all aspects of that environment influence how effectively quality 
management can be carried out, some are critical to its success and deserve explicit 
mention in this Quality Assurance Framework. Program managers are helped in fulfilling 
their program objectives through a series of measures aimed at creating an environment 
and culture within the Agency that recognizes the importance of quality to the Agency’s 
effectiveness. These measures include the recruitment of talented staff and their 
development to appreciate quality issues and an open and effective network of internal 
communications. They include explicit measures to develop partnerships and 
understandings with the Agency’s suppliers (especially respondents). Finally, they also 
include programs of data analysis and methodological research that encourage a search 
for improvement. 


10.1. Partnership with Suppliers 


Suppliers of the raw data upon which all Statistics Canada products are based are 
fundamental contributors to the quality of data and information. Particular attention is 
paid in following-up on respondent complaints. Questionnaires are tested to ensure 
minimal intrusion on privacy, and to respect public sensitivities and overall social 
acceptability. 


Cooperative arrangements with suppliers (for example households, businesses, holders of 
administrative records) are pursued through: 


° a respondent relations program 

° a response burden management program 

° bilateral committees (e.g., with Canada Customs and Revenue Agency) 
° engagement with the small business community 

° a small business ombudsman 

° electronic reporting initiatives 

° recognition of respondents in publications 


These are supported by the Policy on Informing Survey Respondents (Policy 1.1), and 
through the activities of Data Access and Control Division, effective use of 
administrative data, effective questionnaire design as prescribed by the Guidelines for 
Questionnaire Testing and Evaluation, and a tradition of acute awareness of and respect 
for the confidentiality of data. 
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10.2. Recruitment and Training 


The primary means by which Statistics Canada meets its objectives is through the 
knowledge, experience and motivation of its staff. Staff must not only be technically 
expert but also aware of quality issues, and able to develop and implement practices and 
methods to meet quality objectives. The availability of qualified and talented staff to 
programs and projects is therefore critical. The recruitment and development of such staff 
is carried out with the aid of the following initiatives: 


° entry level recruitment and development programs for major occupational 
groups; 

° generic competitions for all management levels; 

° mentoring programs; 

° comprehensive career stream plans for major groups; 

° a training policy and framework; 

° the Statistics Canada Training Institute offering flagship and specialized 
courses; 

: Certificate programs/University Certificate programs; 

° a corporate assignment program; 

° development programs for managers; 

° an Awards and Recognition program. 


A full description of the Agency’s Human Resources Management framework can be 
found in Human Resources Development at Statistics Canada (1997). 


Internal communications are also an important means for maintaining staff awareness of 
quality issues, initiatives and requirements. A variety of communication vehicles are in 
place to provide employees with information and to seek employee feedback on how to 
improve programs and the organization environment. 


10.3. Data Analysis and Research and Development 


Statistical agencies exist to provide current statistical information, but their long term 
survival depends on their ability to adapt and evolve. To do this, they need to continue 
support of analysis, innovation and experimentation as one means of maintaining their 
professional and operational infrastructure. 


Statistics Canada conducts and promotes data analysis to engender an awareness of data 
quality issues from both a developer’s and a user’s perspective. Through analysis the 
Agency not only adds to the information value of its outputs but also promotes pride in 
the relevance, accuracy and coherence of its statistical data. Conducting data analysis 
allows staff to obtain broader contacts and experience. It allows staff to enhance or 
broaden their skills and enhances opportunities for recruiting talented, creative staff. 
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Research and development of a statistical, subject matter, informatics or operational 
nature helps to achieve high quality and to create a culture of quality improvement, in 
addition to yielding efficiency gains. Research and development is carried out by program 
and infrastructure staffs and through collaboration with external consultants and 
researchers. 


11. Conclusion 


Statistics Canada’s quality assurance framework consists of a wide variety of mechanisms 
and processes, acting at various levels throughout the Agency’s programs and across its 
organization. The effectiveness of this framework depends not on any one mechanism or 
process but on the collective effect of many interdependent measures. These build on the 
professional interests and motivation of the staff. They reinforce each other as means to 
serve client needs. They emphasize the Agency’s objective professionalism, and reflect a 
concern for data quality. While the overall framework is inevitably presented in this 
paper as a set of separate components, the important feature of the regime is the synergy 
resulting from the many players in the Agency’s programs, operating within a framework 
of coherent processes and consistent messages. 


The following diagram on the next page attempts to encapsulate this quality assurance 
framework, “at a glance”, as a unified entity, while reminding the reader of the ultimate 
objects of the framework, the continuity and interconnectedness of its processes, and the 
ongoing need for development and improvement within the framework. 
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